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Reviewers have recognized that, in the general evaluation literature, the typical 
program evaluation utilizes poor methodologies practices (Upsey, Crosse. Dunkle. Pollard & 
Stobait. 1965). In the field of education, methodological criticisms have been substantial. 
Experimental design traditionally has received the most attention (Cook and Campbea, 1979). 
This methodological issue dominates most other technical considerations in evaluation 
textbooks (e.g.. Rossi and Freeman. 1972). In general, the quaJty of experimental design in 
field settings throughout the social sciences has been criticized (Upsey. et aJ., 1965). 

This paper discusses the findings of some recent research examining the 
implementation of research design in the domain of treatments for juvenle delnquency 
(Potard. 1990). The major thrust of this research is that the qualty of research design 
appears to depend upon the conflicting and contraolctory influences of a variety of evaluation 
stakeholders. Further, through a process of negative feedback In a manner analogous to a 
thermostat, the evaluation stakeholders eventually constrain the research design within a 
narrow range of methodological rigor. The findings of this research are then applied to 
understand the implementation of an evaluation of the Here's Looking at You, 2000 program, 
an evaluation that is currently underway in California. The decision making and negotiation 
process leading up the the final formulation of the research design in the Here's Looking at 
You, 2000 evaluation illustrates the findings of this research. 

The Implementation of Research Design 
Many evaluators still routinely employ a^-experknental designs. Upsey, et al. 
(1965). in a random sample of 175 published program evaluations conducted between 1978 
and i960, found that 48% of al evaluations utilized weak quasi-experimental designs and 16% 
utilized pre-experimentaJ designs that did not incorporate any form of a comparison or 
control group. While circumstances may limit design options, Boruch and Wothke (1965) have 
pointed out that fully randomized experiments have been successfully implemented in quite 
diverse field environments. Unfortunately, program evaluators still most often rely upon the 



weaker designs. There are probably numerous factors that Urn* the implementation of 
randomized experimental designs. One significant factor may be that the evaluation literature 
has not provided wed defined procedures for how to implement a randomized design in a field 
treatment setting. Boruch and Wothke (1985) suggest that this weakness leads to a falure 
by most evaiuators to seriously consider implementation of a randomized design. Instead, 
there is a too easy acceptance of alternative methods. For example. Boruch and Wothke 
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(1965) remarked that many research design textbooks (e.g. Kirk. 197% Winer. 1971) merely 
state that randomization should take place, but never give concrete procedures for how 
randomization can actualy be conducted. 

There are also substantial poHtJcaJ and ethical questions that must be answered for 
evaluation stakeholders in any evaluation setting that is employing randomization. Individuals, 
not well versed in the logic of research design, can view randomization as arbfcrary. unfair and 
unethical because it seems to be a denial of services to people in need. These questions, If not 
fully answered. wHJ often frustrate the attempt to randomize subject assignment. In what is 
becoming a classic example of this, Sherman and Berk (1964, 1965) Invested an extensive 
amount of time and effort convincing focal authorities that the randomization of arrest 
procedures by police when called to domestic violence situations was both practical, ethical, 
and beneficial to al parties. They encountered substantial resistance, both from the poUce and 
other parties, but eventually were able to implement their plans. It was only because of their 
extensive efforts and persistence that this study was Implemented as it was. 

There are several other examples of randomization in turbulent field settings (see 
Boruch and Wothke. 1965). The common features of these examples are a strong 
commitment to the value of raixtomizatlon by aH parties, a strong and continuing presence of 
the evaluator throughout the Me of the evaluation, and a willingness to educate those 
individuals who wll actually implement the evaluation as to the rationale for randomization. 

In summary, the details of how evaiuators select and implement specific research 
designs are not well known. While the literature on this topic is increasing, no systematic 



studies have previously been completed on this topic. What is available suggests that 
numerous poHtical and social factors influence design choice. 

The Present Research 

Recent research, based on the author's dissertation (Poiard, 1990), provides some 
Insight into many of these questions about how research design is implemented In evaluation 
settings. This research was conducted as part of a large NIMH sponsored meta-analysis of 
the juvenile delinquency treatment evaluation Mterature (Upsey, 1990). In the dtesertatton 
research, 269 evaluation studies of the efficacy of treatments for Juvenile desnqusncy were 
included in a meta-analysis. The primary goals of the otesertatioo metaanalysis were: 1) to 
determine how the social and poWcaJ context of the evaluation affected several Important 
methodological variables in the evaluation, including the method of subject assignment; and 2) 
to determine how the methodological characteristics of the evaluations, as they were 
eventually implemented, eventualy Impacted the reported treatment effectiveness. This latter 
analysis is not reported here but is available from the author (Pollard, 1990). 

In the following results, the degree to whten the method of subject 
assignment could be predicted by variables coding for the poWcal and social context 
of the evaluation (cased the Study Context variables) was examined. (The Study 
Context variable are listed in Table 1.) Looking at the resuKs on Table 1 for the 
variable Method of Subject Assignment, about 40% of the studies used randomized 
methods of subject assignment, while 26.8% of the studies used strong quasi- 
experimental designs (e.g., matched subject designs, equated groupwise), while the 
remainder (24.9%) used weaker quasi-experimental designs (convenience samples of 
volunteers. Intact groups, etc.). Studies with pre-pcst designs were not included art 
the research . This analysis investigated the extent the evaluate** choice of method 
of subject assignment was predictable from other characteristics of the evaluation 
that presumably predate the evaluator's choice of design. In other words, do 



characteristics of the early evaluation context in some way constrict or influence the 
researcher's choice of method of subject assignment? 

in Table 2. the summary results of the regression analysis predicting Method 
of Subject Assignment from several Study Context variables, one missingness 
variable and one coder confidence variable are presented. In Table 2 the criterion 
variable is feted at the top with the number of valid cases utized in this analysis. The 
meaning of the direction of the correlations is also provided at the top ofJhe table. 
The predictor variables, and their order of entry, are provided In the next section, 
included in this section are the correlation coefficients (r) between the Individual 
predictor variables and the criterion, as wen as the final beta values and their 
associated t-tests. If the beta value was not significant at least to the p < 15 level, 
the t-test value was not entered into the table. In the final section of the table, in the 
Summary of Results section, the R 2 change and R^^ values ore provided, along 
with F ratios and significance levels for each set of variables. 

A missingness variable was included in the analysis with Its corresponding 
coded variable if more than 10% of the citations reported missing data on the coded 
variable. In the case of the analysis presented in Taoie 2, only one variable, coding for 
whether the study was conducted at a Criminal Justice Site, mei this criterion. 

The first set of variables entered into the regression equation were the 
variables: Year of Pubiterton and Academic Affiliation of Senior Author. The second 
set of variables entered were the remainder of the Study Context variables. In the 
third and final step of this regression, only a single variable was entered. Coder 
Confidence in Method of Subject Assignment. This variable represented how 
confident the person coding the evaluation report for the meta-analysis project felt in 
their judgement about the Method of Subject Assignment. In terms of the causal logic 
outlined above, each successive set of variables was presumed to be potentially able 



to influence the criterion variable and the predictor variables that entered later into 
the regression equation, but rx>t the variables that have already entered the equation. 

In Table 2, after entry of the first two variables, R 2 ■ .029. p -021. This 
means that approximately three percent of the variance of the criterion variable 
Method of Subject Assignment could be accounted for by the two variables Year of 
Publication and Academic Author Affiliation. 

After the introduction in Step 2 o* the remainder of the Study Context 
variables. R^rotal ■ - 194 > (P^OOi). This represents the total amount of variance 
accounted for by all of the variables entered into the analysis to that point. The 
amount of variance accounted for by the second "rfable set was R 2 Change-' | 64 (p 
= .0001). This is calculated simply bv taking the Total R 2 at that point, and 
subtracting the Toial R 2 calculated at the previous step. In this case R 2 Change - 
.194 - .030 - .<64. 

Finally, in Step 3, the Coders' Confidence in Method of Subject Assignment 
was entered into the regression. This variable, entered alone, made a significant 
contribution to the analysis (R^ange = 040, p <001). The Total R 2 after the entry 
of this final variable was .233, p<.001. 

At the individual variable level, six variables were significant predictors of the 
criterion variable, as assessed by their individual beta weights and associated Meat 
statistic. Academic Author Affiliation, E valuator Involvement with the Treatment, 
whether the evaluation was conducted with a Demonstration Program , Criminal 
Justice Site, and Coder Confidence in Method of Subject Assignment were all 
positively correlated wftn Method of Subject Assignment. This means that the author 
reporting an academic affiliation, the evaluator being heavily involved with the 
treatment, the treatment being described as a demonstration program, the treatment 
being conducted at a criminal justice site, and the coders' confidence in the method of 
subject assignment were all associated with the presence of a randomized method of 



subject assignment. Conversely. Mandatory Treatment Participation was negatively 
correlated with Method of Subject Assignment, meaning that the juveniles being 
forced to partiCpate in the experimental treatment was associated wfch weaker 
methods of subject assignment. The Interaction between Evaluator Involvement and 
Demonstration Program showed a negative, non-stgnWcant relationship with 
Method of Subject Assignment. 

At the IndMduaJ variable level, most of the above effects intJasy appear 
straight forward in their explanation. Academic evaluators are *ely to adopt a more 
rigorous method of subject assignment because of tne* training and research 
activities. The control offered the evaluator by conducting the study at a CJ site 
would make the adoption and implementation of a randomized procedure logtoticaly 
more feasMe. Also, naturaty enough, evaluators interested enough in the 
program/treatment delivery may be more wMng to adopt a careful approach to 
evaluation. They also are more Ikely to be in a position to approve of and enforce a 
good randomization procedure. Consistent with this explanation, it may be that 
evaluators who are independent of the program do not have the political "clout" to 
generate the approval or implementation of a randomized procedure, which is typically 
more difficult to get program stakeholders to approve. It is not clear is why 
Mandatory Treatment Participation, which again should offer the evaluator greater 
Influence and control in the evaluation setting, was negatively correlated with 
randomized methods of subject assignment. It Is posstote that, because of the 
seriousness of offenses associated with juveniles m institutional settings, more outside 
stakeholders were involved in the formation of the treatment group (e.g.. law 
enforcement officials, judges, institutional officials). These stakeholders may be less 
wiling to allow randomization to occur in settings where strict control over the 
juveniles and their services is maintained. (The next analysis, looking at control group 
activity, provides results supporting this interpretation.) 
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Finally, coder confidence regarding the method of subject assignment utilized 
in the study made its own significant contribution to the predictability of Method of 
Subject Assignment. One possible Interpretation of this findings is that randomization 
was associated with better quality (or more enthusiastic) evaluation reporting. 
Perhaps evaluation authors are more wiing to be exploit on this variable when they 
have utilized a randomized procedure. Thus, the coders were simply more Ikery to be 
more confident of the design when the evaluation report was more expNdt. 

However, interpretation of the Individual variable effects does not provide a 
complete view of these results. It Is important in this analysis to compare the final 
beta weights of the individual variables with their associated correlation coefficients 
(r). Generally speaking, the final beta weights are smaler than the original correlation 
coefficients between the predictor variables and the criterion. This is because the 
variance accounted for by the correlation coefficient includes some criterion variable 
variance that is not uniquely associated with the predictor variable. The beta weight, 
on the other hand, represents only that criterion variable variance that is uniquely 
associated with predictor variable. Thus, the beta value is typically smaHer than the 
correlation coefficient. 

The final beta weights reported in Table 2 were not consistent with this 
pattern. Several predictor variables showed beta values either of opposite sign, or 
larger than, their correlation coefficients. When this pattern occurs, this indicates 
that the measurement of the direct relationship between a criterion and a predictor 
variable (the correlation coefficient) Is being "suppressed" by the effects of other 
predictor variables. That is, the effect of the predictor variable (X1) upon the 

criterion (Y) is obscured by the effects of other predictor variables (X2, X3 Xn), 

that are themselves correlated with the first predictor and the criterion. 

Because of the rote that suppression wW play in this and two later analyser,, it 
is worthwhile to elaborate on the concept of suppression. Fa an hypothetical 



example of suppression, suppose that the predictor variable Level of Taxation (X1) is 
positively associated with the criterion variable Government Revenue (Y). and that 
the predictor variable Honesty in Tax Returns (X2) also is positively associated with 
Government Revenue (Y). However, suppose that the Level of Taxation (X1) is 
negatively associated with Honesty in Tax Returns (X2). because as the burden of 
taxation Increases . people are more Hkeiy to find "deductions" to reduce their tax 
HabWy. Further, as honesty dednes, the government might be IncSned to raise the 
taxation level to make up the lost revenue. The relationship between these three 
variables is presented in Figure 1. 

In the hypothetical case represented by Figure 1, the beta weights for the 
predictor variables might wefl show a moderate positive relationship wth the criterion, 
while the correlation coefficients between the predictors and tha criterion would be 
smaller than the beta weights. For example, the Indirect effect of Level of Taxation 
0(1), acting to reduce Honesty in Tax Returns (X2), would have the effect of 
lowering government revenue. Thus, as the Level of Taxation (X1) increases, 
Government Revenue (Y) decreases because of the indirect effect Level of Taxation 
(X1) has through Honesty in Tax Returns (X2), but increases because of the direct 
link between the two variables. In effect, the direct and indirect effects of Level of 
Taxation (X1) counteract each other. Thus, without statistically controMng for the 
value of Honesty In Tax Returns (X2), the correlation coefficient between Level of 
Taxation (X1) and Government Revenue (Y) appears small, a course, the reverse 
could also occur, where the indirect effect and direct effects of Honesty in Tax 
Returns (X2) would act in a similar, antagonistic way upon Government Revenue (Y). 
Given actual values for the correlations between the variables, an equttxium point 
could be calculated which would maximize Government Revenue (Y) through the 
choice of optimal values fa Level of Taxation (X1) and Honesty in Tax Returns (X2). 
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Thte example of suppression has been presented in some detaR because It 
appears that some of the predictor variables influencing Method of Subject 
Assignment act in a similar fashion. Returning to the analysis of Method of Subject 
Assignment, one example of suppression occurs with Criminal Justice Ste and 
Academic Author AffiHatJon. The beta values for these variables indicate that they 
both have a positive relationship with Method of Subject Assignment. However, 
Criminal Justice Site and Academic Author Aflilatton are negadvety correlated with 
each other. If Criminal Justice Site is substituted for Level of Taxation (X1). 
Academic Author Affiliation is substituted for Honesty In Tax Returns (X2), and 
Method of Subject Assignment is substituted for Government Revenue (Y), the 
pattern of relationships, and the suppression effects, are exactly the same. For both 
predictor variables, their indirect effect upon the criterion Method cf Subject 
Assignment would be in the opposite direction of their direct effect reflected by the 
beta weight. This produces the suppression of the correlation coefficient. 

There are other exampiee of variables in mis analysis exhWting the same 
pattern. One way to systematically identify suppressive variable relationship* is the 
following: Any two predictor variables that both exhfc* the same sign for their final 
beta weight, either positive or negative, wil be in a suppressive relationship when the 
predfctor variables are negatively correlated. Conversely, any two predtetor variables 
with opposite signs for their beta weights wli exist In a suppressive relationship when 
there exists a positive correlation between the predictors. 1 

Table 3 presents three correlation matrices. The upper matrix includes orty 
variables that exhibit positive beta values with the criterion Method of Subject 

1 This decision rule does not identify all suppressive variable relationships. Al 
suppressive relationships can be identified by determining when any one of the 
correlations between the two predtetor variables and the criterion, ryt, rv2. or M2, is 
less than the product of the other two correlations. This decision rule described in 
the text has been used m p*i situation for purposes of clarity in understanding the 
analysis of Method of Subject Assignment (SH27). 
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Assignment. In that matrix, aB statistically significant (p< 05) negativt correlations 
between the predictor variables in the matrix are presented. A negative correlation 
indicates that a suppressive relationship exists between the predictor variables. 
(Positive correlations *re omitted for clarity.) The second matrix includes only 
variables that exhibit negative beta values with Method of Subject Assignment For 
this matrix, again only statistically significant negative correlation coefficients m the 
matrix are presented. In the third matrix at the bottom, predfctor variables hiving a 
negative beta weight are listed along the left side of the matrix. Predictor variables 
having a positive beta weight are listed across the top of the matrix. For this 
correlation matrix, a positive corretatmn between the two predfctors is indicative of 
suppression. The missingness variable fa Criminal Justice Sfte and Coder 
Confidence in Method of Subject Assignment have not been included for clarity. 

In Table 3 It can be seen that for the criterion Method of 8ub)ect Assignment 
many pairs of predictor variables exhibit suppression. One interesting example 
involves the predictor variable Interaction between Evaluator Involvement and 
Demonstration Treatment. This variable, initially showing a modest positive 
correlation coefficient with the criterion, shows a final beta weight of approximately 
equal magnitude but of opposite sign (B - -.280, p - .1171). Suppression appears 
between this variable and the predictor variables Evaluator involvement with the 
Treatment and Demonstration Program . These two variables both exhibit positive 
beta weights with the criterion. And, as would be expected because of their role in 
the creation of Interaction between Evaluator involvement and Demonstration 
Treatment, both are strongly and positively correlated with Interaction between 
Evaluator Involvement and Demonstration Treatment. 

Interaction between Evaluator Involvement and Demonstration Treatment 
was created for this research because it was anticipated that special effects might be 
fourd in evaluations where an academic evaluator would be testing an experimental 
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demonstration program of his or her own (Won. m particular, t wn expected that 
this variable might capture evaluations with unusually strong methodological 
characteristics. Judging from these resufta. and contrary to exped it io ns . It appears 
this variable was sometimes essocieted with evaluations where a randomized method 
of subject assignment was not frequentty employed. This resuft might indicate a 
situation were development of the experimen ta l treatment Is the primary objective 
and/pr the maturity of the treatment program is not yet ready to support a rigorous 
evaluation. 

The question remains how should the overal pattern of suppression among 
the predictor variables should be interpreted. As previously noted, suppression is 
often characteristic of homeostatic systems. Homeostasis arises because system 
variables provide negative feedback when the system deviates from a system norm or 
equHfcrium point. One posstoie explanation for widespread pattern of suppression 
with Method of Subject Assignment is that there is an "equWxIum point" for the rigor 
of the research design In many evaluation settings. The context of an evaluation is 
usually made up of several key players, or stakeholders, who are interested in the 
conduct a outcome of the evaluation. In addition, there are important variables such 
as supporting resources and the avaNabity of program dents, that also impact 
implementation of the research design. Evaluation stakeholders often do not have 
common interests in the evaluation, and place conflicting demands on the evaluation. 
For example, policy makers or senior administrators might want to have the most 
unbiased assessment of program effects, while program personnel might want to 
minimize the impact of the evaluation on the program's clients (e.g., by preventing the 
"unfair" denial or postponement of services to some clients tar purposes of 
randomization). Further, stakeholders may have expectations regarding the 
evaluation that are inconsistent with the evaluation resources. 
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In this context, the evaluation enterprtee is simiar to a rxxneostatic system. 
Various sufreholdsra. resources, and program and cHant factors provide negative 
feedback to the evafcator when the evaluation design deviates too signlffcanCy from 
the* expectations or Imtations. The net effect of theaa Influences to that many 
evaluations may reach an e^flbrium poim In the rigor ot the raaaarch design. This 
eoj*tx1um point would be the method 

the sodal, poUdcal, and economic context of the evaluation. From this perspective, 
the frequent suppression evident wth the preolcior variables m this analysis 
represented the negotiated, evolutionary process that occurred in the formation of 
the research design. 

Subject Assignment Procedures m the Were* Looking at You, 2000 Evaluation 
An evaluation of Hera's Looting at You, 2000. currently in progress in California, 
provides a concrete example of the above findngs. The Were* Looking At You. 2000 
curriculum is a comprehensive, K-12 curriculum designed to prevent or refece the student's 
use of alcohol, tobacco, and marijuana (Roberts, Ftomahan & Associates, 1966). Mara* 
Looking* You, 2000 focuses on these three drugs because they are beloved to act as 
-gateway drugs,'' meaning that adolescents typicaly begin their drug careers with the use of 
these drugs prior to beginning use of other Melt drugs. Here* Looking at You, 2000 to 
designed specifically to counteract the known risk factors leading to adolescent drug abuse: 
fan«y history of abuse of alcohol and other drugs, permissive famty attitudes towards 
alcohol, tobacco and other drugs, academic failure, lack of social bonding, peer Influences, and 
the student's favorable attitudes towards drug use. The lesson plans developed by Hen's 
Looking at You, 2000 focus on developing the students internal resources which wM be 
necessary to avoid drug use. SpecMtealy, Here's Looking at You, 2000 1) promotes bonding 
between the student and society. 2) provides students information on the effects of drugs. 
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and 3) provides the students the necessary eke* by which they can resist peer artf societal 
pressures to use ateohol, tobacco and marijuana. 

Hen's Looking* You, 2000 is the most popular drug prevention curriculum In school 
districts in Calfomia (Duerr Evaluation Resources. 1080)- However, other than anecdotal 
evidence, there Is currently Ktle empirical evidence measuring the effectivene ss of the Here* 
Looking af You. 2000 curriculum. Given that a substantial amount of money le being spent by 
CaWornia Schools on incrementation of the Here's Looking at You, 2000 curriculum, there is 
good reason to evaluate whether measurable, positive effects of the program can be 
identified. 

The district in which the Hero's Looking af You, 2000 evaluation ie being conducted te 
ameolumsizediSstriawlmalajgegeogra 

In addition, it has a wide variation in SES among its students. The dtetrtct has been active in 
drug prevention efforts for many years, beginning to develop comprehensive services In the 
early 1960*8 In general, the district is open to the development and evaluation of drug 
prevention efforts perceived to be beneficial to its students. 

Several stakeholders in the district played a significant role in the development of the 
evaluation. However, not all of them were equaly enthusiastic or committed w the conduct 
of the evaluation. Most stakeholders' attitudes towards the prospective evaluation were a 
function of their perceived gains or losses as a result of the conduct of the evakjatkxi. Their 
positions are reflected in Table 4. 

Rm Table 4 It can be seen that there was a relative balance in the position of the 
stakeholders between those who wanted to adopt a rigorous method of subject assignment 
(e.g., randomization of individual students) and those who resisted this approach. Those 
stakeholders who resisted randomization did so primarily because of the disruption and the 
logistical difficulties that would be encountered if indlvkiual students were assigned either to 
receive the Here's Looking st You, 2000 curriculum or would receive no services. Because of 
the amount of classroom time the Here's Looking at You, 2000 curriculum requires, it is no 
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smal Iriconvenience to be randomly puNing students torn classrooms to receive the 

curriculum. 

The reasons for other stakeholders not accepting less rigorous methods of 
subject sssignment primarty focused on the lack of usefulness of the resufts If a poor 
quafty subject assignment procedure was adopted. The evaluate* was not Interested 
in committing a significant amount of time or resources to the project if 
urtnterpretabie or ambiguous results wero unavoidable. Administrator #1 had a 
simftar, research -minded" orientation and eiso was not very interested in the 
evaluation If rnWmum standaids could riot be mairaairted 

Ultimately, the net effect of these forces were that a compromise was 
reached. The Here's Looking at You, 2000 program was presented m 7th grade social 
etudes classes at each of the four midtfo schools in the tfstrfct This was the fkst 
year that the 7th grade social science teachers would present the cumcuktn, and all 
had to be trained prior to their presenting the program. Further, because of 
constraints in the number of substitute teachers who could cover classes of the social 
science teachers as they were being Mined, and because of Mttta 
of trainer time that was available, teachers training m the Here's Looking af You, 2000 
curriculum had to be spread out across the entire year. It was decided to take 
advantage of these facts by randomly assigning the 7th grade social science teachers 
to receive their training ether in September. February, or May of the 1900-90 school 
year. Each of the three 7th grade social science teachers at each middto school was 
randomly selected to participate In one of the three training sessions. Since the 
teachers were to present the Here's Looking at You, 2000 curriculum Immediately 
after the training, this meant that one-third of the 7th grader social science students 
received the program In October, one-third In March, and one-third would not 
receive the curriculum at si in this school year. In short, whse students were not 
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randomly assigned, the teachers were randomly assigned within each or the four 
middle schools. 

The final form of the evaluation design was the end product of several 
competing and conflicting forces within the evaluation setting. It could be said that 
the final form of tiie subject assignment procedure was a compromise among the 
differing expectations and demands of the stakeholders. But more Importantly, the 
final form was a kind of "equHfcrium point" around which the final evaluation plan 
probably could not have significantly dfflered. The various stakeholders, because of 
their "drop dead" criteria, continually provided negative feedback to the negotiation 
process when the proposed plan differed too widely from their expectations. Rom 
this perspective It appears that the most ttefy outcomes for this evaluation were that 
this particular method of subject assignment would be adopted or that the evaluation 
would not be conducted at al. This kind of mutual constraint among the stakeholders 
was similar to what was described in the meta-analysis research. 

Two other broad conclusions emerge from these findings. Thsse concsjstons are both 
consistent with these resufts, yet are somewhat comradfaory with each other. First, given 
that these results provide the evaluator with more insight into the forces surrounding the 
implementation of subject assignment procedures, the evaluator may be able to plan mote 
effectively. Recognizing the probable forces that surround the implementation of an 
evaluation, the evaluator could more effectively negotiate among the various stakeholders, 
eventually producing a more rigorous evaluation methodology. 

The second broad conclusion, as mentioned, is somewhat contradictory to the above 
comments. The evaluator should be realistic as to what he or stw wiN be able to achieve in 
implementation of research design. Trying to force a rigorous method of subject assignment 
In an evaluation whose context wiN not support that effort may be doomed to failure. A 
typical evaluation problem is that what was planned as a rigorous investigation of the 
effectiveness of a treatment degrades into an evaluation that Is neither as rigorous or as 
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informative as was hoped. If the analysis of this paper it correct, an evaluetor trying to 
knpose a rigorous design in a system whose "equfcxlum point" for method of sttoject 
assignment is substantially lower may be at high risk for frustration. 

These comments are not meant to detract from the work of authort such as Boruch 
and Wothke (1985) who argue that more rigorous evaluations coutd be Implemented &an * 
now routinely the case. In fact, their position is quite consistent wth these views. They 
argue that ths evaJustor can enhan^ 

factors in the evaluation setting that are represented m this analysis. Boruch and Wothke 
state that: "Interest and support at higher levels In too*c!y coupled social systems do not 
guarantee randomization at operating levels. Incentives and benefits that offset ths sharkiR 
of control have to be planned to engage participation and to sustain agreements at tower 
levels." (p. 108) Whfcthelangjjageusedtodeecrtoeto 

Afferent than that used by Boruch and Wothke (1965). the results are consistent with the 
conctostons drawn from their experience. The evaluation enterprise. partteuLrty when ft 
concerns the method of subject assignment, is characterized by numerous social and poKteal 
factors net are mutualy antagonistic and that act to counteract each other's influence. Left 
unrestrained, these factors ufttnatety result in a "compromise" in the method of subject 
assignment. This compromise typteafty gives none of the stakeholders all that they want, and 
results in proliferation of evaluation Uudtor with lees than optimal research designs. Boruch 
and Wothke continue: "It Is not easy to design the randomization process so that It Is robust 
against indifference, incornpetence, and corruption. StJI It Is a task that seems worth ths 
effort." (p. 98) 
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Table 1 

PjMritttea tattertca of tha 



RATIO VARIABLES 



Mean 8.D. VeJdN Mies. 



Year of Publication 


1976.76 6.105 


258 


4.1% 


CATEGORICAL VARIABLES: 


Category 


Pet. 


N 


Affiliation of Senior Author 


Other Afflaation 


31 .2 


84 




Academic AfNatton 


62.1 


187 






6.7 


18 


Evaluation Funding 


Not Rinded 


30.5 


82 




Raided 


67.3 


181 




Missing 


2.2 


6 


E valuator Involvement 1 


dependent of Treatment 


33.8 


01 


with Treatment 


Influential, No Dir. Role 


14.0 


40 


Plan-Control-Superviee Treatment 


33.5 


90 




Delivered Treatment 


7.8 


21 




Missing 


10.0 


27 


Program/Treatment Age 


Relatively New 


63.2 


170 


at Time of Research 


Established (2+ yrs.) 


32.3 


87 




Defunct Program 


1.5 


4 




Missing 


3.0 


8 



Is Treatment Part of Regular Program 62.5 168 

Demonstration Program Demonstration Program 34.9 94 

Missing 2.6 7 



Mandatory Treatment 


Participation Voluntary 


3.0 


8 


for Juveniles 


Referred By Family and Friends 


.7 


2 




Refetred By Non-CJ Agency 


13.4 


36 




Voluntary Referral, CJ Agency 


30.5 


82 




Mandatory Referral.CJ Agency 


45.0 


121 




Missing 


7.4 


20 



9 
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Table 1 (Com.) 
n«*rrirthi« atattetta of ttw 
CATEGORICAL VARIABLES'. 



Cttegory 


Pet. 


N 


Mon-CJ Treatment Ska 


50.9 


137 


Mixed CJ and Non-CJ Tx 


5.9 


16 


CJ Treatment 8fte 


28.3 


76 


Missing 


14.9 


40 



Treatment Conducted at 
Criminal Justice Site 



Treatment Conducted in Not Institutional Setting 69.1 186 
Institutional Setting Institutional Setting 25.3 68 

Missing 5.6 15 



Interaction of Academic Regular Program 


62.5 


168 


AfflHation of Author and Demo. Prog.. Evsi mdapen. 


3.3 


9 


Whether Treatment Is Demo. Prog. Evat mfluen. 


2.6 


7 


Demonstration Demo. Prog., EvaL Planned Troet. 


19.7 


53 


Program Demo. Prog.. Eval. Delvered Treatment 


7.1 


19 


Missing 


4.8 


13 


Method of Subject Weak CKiasi^xpertmental 


24.6 


67 


Assignment Strong Quati-Expertmental 


26.8 


72 


Randomized 


46.1 


124 


Missing 


2.2 


6 



Coder Confidence in Method 


Very Low 


.4 


1 


of Subject Assignment 


Low 


2.6 


7 




Moderate 


7.4 


20 




High 


23.8 


64 




Very High 


62.5 


. 168 




Missing 


3.3 


9 
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Table 2 

HUMPH raoraM Wt mfrjji prtHrrtng ttm M 



Criterion Variable: 
(N-263) 



Method of lubtart Aaatgnawnt 

m lndtea*a conateeon w» randomtrx) 
daatgna.) 



Prodictor Varlablaa: 



Em&AL. 



Stapl: 
Step 2: 



YaarofPuMcafon 
Academic AuftorAffiatlon 

Evaluation Funtfng 
Evai. Invohf . In Tiaabnant 



Step 3: 



EvaL tnvetv. x Demo. Prog. 
PiOQran/Tiesment Aqo 
Mandatory Treatment Part. 
Mtarinmaaa Variable. CJ 9Na 
OrkmneUuaaea(CJ) Sfte 
InatMonal Setting 

Coder Com*, m Subject Aesign. 



r 


MB 


t-yilua 


-.003 


-.008 




.170 


.134 


2464* 


.002 


.067 




.329 


.346 


4.196— 


271 


418 


2.032* 


283 


-460 


-1472* 


-.004 


.067 




-.176 


-.191 


-3437** 


-.099 


-.044 




.014 


.175 


2.730** 


-.034 


-.027 




.258 


.208 


3.589*** 



Summary of Reaulte: 



Stap: 
Stapl: 
Slap 2: 
Step 3- 



.029 
.164 
.040 



3.904 
5.689 
12683 



.021 
.001 
.004 



.030 
.194 

433 



3404 

5.479 
6.333 



.021 
.001 
.001 



A p<-15 *pl-05 **p< .01 ***p< .001 
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Table 3 



values. 



Funding 
AfM. Auth. 
EveJ. Invt. 
Ptoq. Age 
Demon. Pr. 

CJ S«e 



Correlation matrix for predtetor variables exhtttng only postlve beta 



Evriua. Aim.* Eval.."* Program Demon.' CJ" 
BttlM Author Involvo. Age PXPJL fill 



.206 



-.311 



.152 -.122 
-.164 
-.145 

.407 



Correlation matrix for predtetor variables exhfcttag only negative beta vah*«. 



Year of Ev. inv x Mand. I net. 
EUflc* Dam EL A H Pj£L"Sfl!taig 



Year of PubfcatJon 
Eval. Invol. X Dem.P. 
Mandatory Tx Part. 
Institutional Setting. 



-.110 



.200 



-.120 



Correlation matrix for predictor variables with positive & negative beta values. 
PnOctort wftft postfve tott vetoes. 



Predfcfors 
w. Megatfve 



Evalua. AffNia.of Eval."* Program Demon.* CJ" 
Pundmq Author loyojy& Agg ElflO* SKa 



Year Pub. 
E.I. x D.P. A 
Mand. Tx.P ' 
Inst. Sett. 



.125 



.162 .663 



.922 



.252 
.368 



tJojg: The signMcanoe level of the predfctor variable^ final beta weight is 

*p< .15 *p£.05 "p<.01 "*p<.001. 
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STAKEHOLDERS 

E valuator (indudtag 
SWRL and WC organizations) 

Administrator #1 



Administrator #2 



AOO Administrators 



School Site Administrators 



ENTHUSIASM DESIRED METHOD OF 

POP EVALUATION ASSIGNMENT METHOD 



High 

High 
Low-Medium 
Medium-High 

Low 



Fun Randomization 



FuN Randomization 



Pre-Post or 

Quasi-Experirmtntal 

Randomization 
or QuasJ-Experimental 

Pre-Post 



"UNACCEPTABLE" 



WeakQuaei ^ perimen ta i 



Weak Quasi Qpsrimentai 



Randomiz a tion by Student 



None 



Randomization by Student 



Teachers 



Low-Medium 



Pre-Post 



Randomization by Student 



Table 4. Evaluation stakeholders, and their respective positions regarding the Implementation of 
the method of subject assignment. 
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